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A  DoD  CCRP/NATO  Collaboration 


This  major  revision  to  the  Code  of  Best  Practice  for 
C2  Assessment  is  the  product  of  a  NATO  Research 
and  Technology  Organisation  (RTO)  sponsored 
Research  Group  (SAS-026).  It  represents  over  a 
decade  of  work  by  many  of  the  best  analysts  from 
the  NATO  countries.  A  symposium  (SAS-039)  was 
hosted  by  the  NATO  Consultation  Command  Control 
Agency  (NC3A)  and  provided  the  venue  for  a 
rigorous  peer  review  of  this  code. 

This  publication  is  the  latest  in  a  series  produced 
by  the  Command  and  Control  Research  Program 
(CCRP)  under  the  auspices  of  the  Assistant 
Secretary  of  Defense  (C3I).  The  CCRP  has 
demonstrated  the  importance  of  having  a  research 
program  focused  on  the  national  security 
implications  of  the  Information  Age.  The  research 
sponsored  and  encouraged  by  the  CCRP  contributes 
to  the  development  of  the  theoretical  foundations 
necessary  to  support  the  Information  Age 
transformation  of  the  Department.  Other  CCRP 
initiatives  are  designed  to  acquaint  military  and 
civilian  leaders  with  emerging  issues  related  to 
transformation.  This  CCRP  Publication  Series  is  a 
key  element  of  this  effort. 


Check  our  Web  site  for  the  latest  CCRP  activities  and  publications. 
This  code  is  also  available  on  our  Web  site. 


www.dodccrp.org 
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Introduction 

Background  and  Aims 

NATO  has  produced  a  Code  of  Best  Practice 
(COBP)1  in  order  to  facilitate  high  quality 
assessment  in  the  area  of  Command  and  Control 
(C2).  The  COBP  is  the  product  of  international 
collaboration  drawing  together  the  operational  and 
analytical  experience  of  leading  military  and  civilian 
defence  experts  from  across  the  NATO  nations.  The 
COBP  enhances  the  understanding  of  best  practice 
and  outlines  a  structured  process  for  the  conduct  of 
operational  analysis  for  C2,  which  is  the  core 
capability  of  Information  Age  defence  and  security. 

The  command  and  control  aspects  of  military 
capability  are  difficult  to  assess.  Use  of  the  COBP 
will  increase  the  likelihood  of  quality  products  that 
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are  complete,  relevant,  transparent,  credible,  and 
authoritative.  In  particular,  the  COBP  will: 

•  Support  broadening  of  the  analysis  process; 

•  Support  effective  use  of  analysis  in  direct  support 
of  operations; 

•  Improve  the  quality  and  coherence  of  business  cases; 

•  Reduce  risk  and  cost  of  the  planning, 
preparation,  analysis  and  presentation  of 
supporting  documentation; 

•  Provide  a  methodology  acceptable  to  both 
analysts  and  decisionmakers. 

The  COBP  facilitates  effective  structuring  of  the 
analytic  process.  It  describes  a  framework  that  can 
be  used  to  design,  execute,  review,  and  deliver  high 
quality  C2  assessments  that  include  all  key  factors 
and  stakeholders.  Use  of  the  COBP  should  be 
regarded  as  a  community  standard  for  all  levels  and 
scales  of  assessment  studies. 

Definition  of  C2 

C2  has  been  defined  by  NATO  as  Military  Function 
01 :  “The  Organisation,  Process,  Procedures,  and 
Systems  necessary  to  allow  timely  political  and  military 
decisionmaking  and  to  enable  military  commanders 
to  direct  and  control  military  forces.”2  C2  systems  are 
further  defined  in  NATO  documents  to  include: 
headquarters  facilities,  communications,  information 
systems,  and  sensors  &  warning  installations.3  More 
recently,  the  term  C2  has  referred  to  the  collaborative 
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and  consultative  processes  that  are  an  inherent  part 
of  coalition  operations. 

For  the  purposes  of  the  COBP,  the  term  C2  is 
intended  to  be  an  umbrella  term  that  encompasses 
the  concepts,  issues,  organisations,  activities, 
processes,  and  systems  associated  with  the  NATO 
definition  of  C2  as  well  as  the  other  terms 
enumerated  above. 

Why  is  C2  Special? 

C2  is  special  because  it  explicitly  involves 
representation  of  the  human  component.  The  focus 
of  military  research  and  analysis  has  predominantly 
been  on  the  physical  domain.  C2  deals  with 
distributed  teams  of  humans  operating  under  stress 
and  in  a  variety  of  other  operating  conditions.  C2 
problems  are  thus  dominated  by  their  information, 
behavioral,  and  cognitive  aspects,  which  have  been 
less  well  researched  and  understood.  This  focus 
creates  a  multidimensional,  complex  analytic  space 
that  involves  multi-sided  dynamics  including 
friendly,  adversary,  and  other  actors,  action- 
reaction  dynamics,  and  tightly  coupled  interactions 
among  elements  such  as  doctrine,  concepts  of 
operations,  training,  materiel,  and  personnel. 

C2  issues  are  difficult  to  decompose  and 
recompose  without  committing  errors  of  logic. 
Moreover,  the  composition  rules  by  which  the 
various  factors  inherent  to  C2  interact  are  poorly 
understood  except  in  arenas  that  have  been 
previously  studied  in  detail.  Finally,  the  C2  arena 
is  weakly  bounded,  with  issues  that,  although  on 
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initial  examination  appear  quite  finite,  often  prove 
to  be  linked  to  very  high-level  factors. 

The  COBP  is  intended  to  assist  the  community  in 
dealing  with,  and  overcoming,  the  barriers  to  effective 
C2  assessment. 

Structure  of  the  Guide 

The  rest  of  this  guide  is  structured  to  follow  the  chapters 
of  the  COBP  and  abstracts  definitions,  principles,  and 
key  points  of  guidance. 

Preparing  for  Success: 

Assessment  Participants, 
Relationships,  and  Dynamics 

The  issues  associated  with  assessment 
participants,  relationships,  and  dynamics  must  be 
addressed  in  order  to  establish  a  sound  foundation 
for  a  successful  assessment.  As  an  initial  step,  the 
individuals  who  are  involved  in  the  study  must  be 
identified,  the  relationships  among  the  participants 
must  be  understood,  and  a  common  understanding 
of  the  study’s  goals,  objectives,  scope,  and 
administrative  aspects  (e.g.,  schedule,  budget) 
must  be  established  and  documented.  An 
assessment  team  must  be  assembled  that 
manifests  the  necessary  skills  and  experience,  and 
the  broad  plan  of  attack  must  be  formulated. 
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Assessment  Participants 

Early  in  the  study,  the  assessment  team  should 
identify  which  individuals  and  organisations  should 
perform  key  roles  in  the  assessment.  The  roles  of 
interest  include,  inter  alia,  assessment  team 
members,  decisionmakers  or  problem  owners, 
stakeholders,  bill  payers,  existing  and  future  study 
teams,  peer  reviewers,  data  providers,  assumption 
providers,  and  data  collectors.  It  is  prudent  for  the 
assessment  team  to  map  the  roles  onto  the 
individuals  and  organisations  involved  and  to 
understand  their  interrelationships. 

Relationships  Among  Participants 
and  the  Conduct  of  the  Assessment 

The  relationship  among  the  assessment  team,  the 
key  sponsor,  and  the  stakeholders  is  of  paramount 
importance  and  will  influence  the  course  and 
success  of  the  effort.  The  assessment  team 
should  be  aware  that  the  diverse  participants  may 
have  divergent  perspectives  and  agendas. 

The  assessment  team  should  undertake  the 
following  actions: 

•  Maintain  long-term  relationships  (including  an 
ongoing  dialogue)  with  the  sponsor  and  stakeholder 
organisations.  This  will  yield  substantial  dividends  in 
the  form  of  easier  communication,  greater  trust,  and 
stronger  support. 

•  Generate  a  concise,  agreed-to  Terms  of 
Reference,  covering  goals,  scope,  products, 
schedule,  and  resources. 
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•  Find  out  at  an  early  stage  in  the  project  what  the 
products  of  the  study  are  to  be  used  for  by  the 
sponsor  and  stakeholder  organisations. 

•  Early  in  the  study,  agree  on  a  common  language 
and  develop  a  study  glossary.  This  should  evolve 
during  the  course  of  the  study. 

•  Perform  a  rapid  first  pass  of  all  the  phases  of  the 
project  to  help  establish  the  approximate  budget 
requirements,  especially  in  large  C2  projects. 

The  assessment  team  should  establish 
strategies  to  address  shortfalls  in  the  event  the 
sponsor’s  resources  are  limited  to  a  level  below 
what  is  required  to  support  a  quality  study. 

Assessment  Team 

•  The  precise  skills  and  experience  required  by 
the  assessment  team  must  be  established 
following  initial  problem  formulation.  The 
assessment  team  must  be  interdisciplinary. 

•  As  an  illustration,  the  following  are  representative 
skills  and  experience  needed  for  the  core  team 
and  consultants  and  part-time  team  members  for 
a  recent,  complex  C2  assessment: 

•  Core  Team:  project  management;  OR/OA 
skills;  cross  military  experience;  human  science 
and  organistational  theory;  data  collection. 

•  Consultants  and  Part-time  Team  Members: 
military;  training  and  exercise  planning; 
communications  and  information  systems 
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expertise;  human  computer  interface 
expertise;  OOTW  related  issues. 

•  Sufficient  time  and  a  facilitating  process  should 
be  built  into  the  project  plan  for  the  group  of 
individuals  to  coalesce  into  a  team. 

Assessment  Process 

•  The  assessment  process  is  non-linear  and  iterative. 

•  The  assessment  team  must  realize  that  all  of 
the  elements  of  the  C2  assessment  are 
interrelated.  Hence  Problem  Formulation, 
Solution  Strategy,  Measures  of  Merit, 
Scenarios,  Human/Organisational  Factors, 
Models  and  Tools,  Data,  and  products  are  all 
interdependent  (See  Figure  1). 

•  Peer  review  is  a  necessity,  not  a  luxury. 

Problem  Formulation 

Effective  problem  formulation  is  fundamental  to  the 
success  of  all  assessments,  but  particularly  in  C2 
assessment  because  the  issues  are  often  ill- 
defined  and  complex,  involving  many  dimensions 
and  a  rich  context.  The  assessment  team  must 
perform  problem  formulation  carefully  and 
understand  its  underlying  principles.  There  are 
principles  of  problem  formulation  that  apply  to  C2 
assessment  in  general,  some  of  which  are 
particularly  important  to  C2  assessment  for  OOTW, 
in  particular.  Drawing  on  these  principles,  one  can 
characterize  the  problem  formulation  process  and 
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the  products  that  are  to  be  produced  in  the  problem 
formulation  phase. 


Figure  1 .  C2  Assessment  Process 


Definition  of  Problem  Formulation 

•  Problem  formulation  is  an  iterative  process  that 
evolves  over  the  course  of  the  study.  Iteration 
is  essential  even  for  small  studies  or  where 
time  is  short — it  will  save  time  later  and  help 
ensure  quality. 

•  Problem  Formulation  is  fundamentally  a  social 
process  of  developing  a  shared  understanding. 

•  The  context  of  the  study  includes  geopolitical 
context;  political,  social,  historical,  economic, 
geographic,  technological  environments;  actors; 
threats;  aims  and  objectives  of  the  assessment, 
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including  the  decisions  to  be  supported;  generic 
C2  issues;  relevant  previous  studies;  and 
stakeholders  and  their  organizational  affiliations. 

•  The  aspects  of  the  problem  include  issues  to  be 
addressed;  assumptions;  high-level  Measures  of 
Merit  (MoM);  independent  variables  (controllable 
and  uncontrollable);  constraints  on  the  values  of 
the  variables  (domain  and  range);  time 
constraints  on  delivery  of  advice  to  the 
decisionmaker;  and  whether  this  is  a  single 
decision  or  (possibly  one  of)  a  chain  of  decisions 
to  be  made  over  time. 

•  The  problem  is  not  formulated  until  the 
assessment  team  has  addressed  each  aspect  of 
the  problem. 

Principles  of  Problem  Formulation 

•  Proper  problem  formulation  takes  substantial 
time  and  effort! 

•  Explicit  problem  formulation  must  precede 
construction  of  concepts  for  assessment  or 
method  selection. 

•  The  assessment  team  must  have  an 
understanding  of  the  decisions  to  be  supported 
by  the  assessment  and  the  viewpoints  of  the 
various  stakeholders  to  clarify  the  study  issues. 

•  Problem  formulation  must  not  only  provide 
problem  segments  amenable  to  analysis,  but 
also  a  clear  and  valid  mechanism  for  meaningful 
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synthesis  to  provide  coherent  knowledge  about 
the  original,  larger  problem. 

•  Problem  formulation  must  be  broad  and  iterative 
in  nature,  accepting  the  minimum  of  a  priori 
constraints  and  using  methods  to  encourage 
creative  and  multi-disciplinary  thinking. 

•  The  problem  formulation  process  should  not 
focus  prematurely  on  subsets  of  the  problem. 

•  Practical  constraints  such  as  data  availability, 
study  resources  (including  time),  and  limitations 
of  tools  should  be  treated  as  modifiers  of  the 
problem  formulation  rather  than  initial  drivers. 
Such  constraints  may,  in  the  end,  drive  the 
feasible  solutions,  but  it  is  important  to  recognise 
this  as  a  compromise  rather  than  an  ideal. 

•  Problem  formulation  should  address  risk  from 
multiple  perspectives.  Risk  analysis  techniques 
should  be  used  to  directly  explore  options  to 
mitigate  risk. 

Principles  Particularly  Appropriate  for 
OOTW  02  Assessments 

•  Problem  formulation  must  address  the 
geopolitical  context  of  the  OOTW  problem  and 
seek  to  identify  the  “broad”  C2  issues  contained 
within  the  Terms  of  Reference  for  the  study. 

•  OOTW  C2  assessments  often  involve  policy- 
related  impacts  outside  the  context  of  a  particular 
military  operation.  Therefore,  MoM  hierarchies 
must  contain  Measures  of  Policy  Effectiveness. 
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•  The  assessment  team  must  have  an  historical 
perspective  to  understand  OOTW  issues 
because  social  conflict  and  structures  often  have 
roots  far  back  in  history. 

•  The  assessment  team  must  have  access  to 
subject  matter  experts  from  a  broad  range  of 
disciplines  (e.g.,  social  scientists,  historians,  and 
regional  experts  in  OOTW  assessment). 

Problem  Formulation  Process 

•  The  process  begins  with  the  sponsor  presenting 
the  assessment  team  with  a  problem  to  assess 
and  an  articulation  of  broad  constraints. 

•  During  the  early  stages  of  problem  formulation  the 
assessment  team  should  quickly  cover  the  whole 
assessment  process  and  produce  an  initial  formulation. 

•  The  assessment  team  must  identify  what  it 
perceives  as  the  real  issues  to  address.  The 
team  must  engage  in  a  dialogue  with  the  key 
sponsor  and  stakeholders  to  get  “buy  in”  for 
these  issues. 

•  In  dealing  with  fuzzy  or  uncertain  boundaries,  the 
assessment  team  should  explore  and 
understand  the  significance  of  each  proposed 
boundary.  The  assessment  team  should  keep  an 
open  mind  during  the  early  stages  of  problem 
formulation  about  where  the  boundaries  lie  and 
their  dimensional  nature. 

•  While  clear  definitions  and  hard  conceptual 
boundaries  are  ultimately  necessary  in  order  to 
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create  a  manageable  problem  space,  the 
assessment  team  should  avoid  coming  to 
closure  prematurely. 

•  Identification  of  high-level  MoM  should  start  with 
ideal  measures  of  the  desired  benefits  before 
considering  what  can  be  practically  generated  by 
analysis  (the  latter  may  force  the  use  of 
surrogate  MoM,  but  these  must  be  clearly  related 
to  the  desired  measures). 

•  The  assessment  team  should  identify,  develop  (if 
necessary),  and  apply  appropriate  tools  to 
support  problem  formulation.  Representative 
tools  and  techniques  include:  techniques  for 
supporting  expert  elicitation,  influence  diagrams, 
causal  maps,  system  dynamics  models,  and 
agent-based  models. 

Problem  Formulation  Products 

An  iteration  of  the  problem  formulation  process  can 
be  said  to  be  complete  when  the  following  is 
accomplished  and  documented:  the  “real”  question 
to  be  answered  is  known;  the  assumptions  have  been 
articulated;  and  the  high  level  MoM,  the  independent 
variables,  and  the  constraints  associated  with  the 
variables  have  been  identified. 

Solution  Strategies 

The  Problem  Formulation  phase  clarified  “what”  is 
to  be  achieved  during  the  assessment.  The  Solution 
Strategy  phase  must  transform  this  understanding 
into  “how”  these  goals  and  objectives  are  to  be 
achieved.  Even  if  the  way  ahead  seems  clear,  the 
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articulation  of  a  formal  solution  strategy  is 
necessary.  Thus,  the  assessment  team  must 
understand  the  definition  and  principles  of  the 
solution  strategy,  the  process  by  which  it  is 
developed,  and  the  products  that  it  produces. 

Definitions  and  Principles 

A  solution  strategy  consists  of  the  specification  of  a 
set  of  sequential  and  parallel  analytical  steps 
(documented  in  the  Study  Plan),  often  involving 
several  methodologies  and  tools.  The  solution 
strategy  is  designed  to  begin  with  what  is  known,  and 
by  execution  of  the  specified  steps,  leads  to  what 
one  desires  to  know — an  illumination  of  the  issues. 

•  The  Study  Plan  consists  of  two  inter-related 
parts — the  formulated  problem  (the  What)  and 
the  solution  strategy  (the  How). 

•  The  solution  strategy  should  not  be  designed 
before  an  initial  pass  through  the  problem 
formulation  process  and  the  problem  formulation 
products  are  available  to  the  team. 

•  The  assessment  team  should  always  remember 
the  inherently  iterative  nature  of  the  process. 

Developing  a  Solution  Strategy 

•  The  solution  strategy  should  strike  an  artful 
balance  between  what  the  team  would  like  to  do 
and  what  is  possible  to  do,  given  the  state  of  the 
art,  available  data,  tools,  schedule,  and 
resources  available. 
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•  The  team  should  first  elaborate  on  the  measures 
that  are  to  be  evaluated  in  the  study.  Using  these 
measures  and  consideration  of  human  and 
organizational  factors,  a  conceptual  model  of  the 
analysis  should  be  developed  and  refined  as 
data  requirements,  methods  and  tools,  and 
scenarios  are  selected  for  the  analysis. 

•  Frequently,  a  solution  strategy  becomes  complex, 
thereby  requiring  the  team  to  decompose  the 
problem  into  parts,  each  of  which  requires 
assessment  with  its  own  set  of  tools. 

•  Taken  together,  the  solution  strategy  must 
include  the  MoMs,  relevant  human  and 
organizational  factors,  specification  of  scenarios, 
data  collection  requirements,  and  methods  and 
tools  to  be  used  in  the  analysis. 

Planning  Documents 

•  The  solution  strategy  is  documented  in  a  Study 
Plan  that  links  the  problem  formulation  and 
solution  strategy  together  in  one  plan. 

•  The  Study  Plan  should  be  developed  in  an 
iterative  fashion,  applying  guidance  and 
feedback  received  from  the  study  sponsor  and 
the  study  stakeholders. 

•  The  Study  Plan  is  typically  approved  and  signed 
by  the  sponsor  (decisionmaker).  Often,  the  Study 
Plan  is  supported  by  a  Study  Management  Plan 
to  guide,  manage,  and  coordinate  the  efforts  of 
the  effort. 
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•  The  Study  Management  Plan  may  have 
subordinate  plans,  to  include  an  Analysis 
Plan(s),  Modeling  and  Simulation  Plan,  Data 
Collection  Plan,  Configuration  Management 
Plan,  Quality  Assurance  Plan,  Review  Plan, 
Deliverable  Plan,  Security  Plan,  a  Study  Risk 
Register,  and  Glossary. 

Measures  of  Merit 

Definitions 

MoM  is  a  generic  term  to  encompass  different  classes 
of  measures.  The  measures  are  defined  in  hierarchical 
levels  related  to  each  other,  each  in  terms  of  its  own 
boundary.  An  orchestrated  set  of  MoMs  is  typically 
required  for  C2  assessments.  The  COBP  has  adopted 
the  following  hierarchical  set  MoMs: 

•  Measures  of  Policy  Effectiveness  (MoPE)  that 
focus  on  policy  or  societal  outcomes  (e.g., 
transition  measures,  which  focus  on  the  progress 
in  the  transfer  of  responsibilities  to  a  follow-on 
military  force  or  civil  agency,  and  normality 
indicators  which  measure  the  quality  of  life  of  the 
civilian  population). 

•  Measures  of  Force  Effectiveness  (MoFE)  that  focus 
on  how  a  force  performs  its  mission  or  the  degree 
to  which  it  meets  its  objectives  (e.g.  loss  exchange 
ratios,  combat  effectiveness,  number  of  targets 
destroyed  and  desirable  adversary  behaviour). 

•  Measures  of  C2  Effectiveness  (MoCE)  that  focus 
on  the  impact  of  C2  systems  within  the 
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operational  context  (e.g.,  time  to  develop  a 
Course  of  Action,  ability  to  provide  information  in 
required  format,  impact  of  information 
operations,  and  planned  quality). 

•  Measures  of  Performance  (MoP)  that  focus  on 
internal  system  structure,  characteristics,  and 
behaviour  (e.g.,  time  to  recognize  an  event, 
correctness  of  perception  and  system  reliability). 

•  Dimensional  Parameters  (DP)  that  focus  on  the 
properties  or  characteristics  inherent  in  the  C2 
system  (e.g.,  bandwidth,  data  access  times, 
cost,  and  size;  characteristics  of  organization 
forms,  attributes  of  personnel). 

Principles 

•  A  multi-faceted  and  sometimes  multi-phased 
approach  is  recommended,  as  no  single 
measure  or  methodology  exists  to  satisfactorily 
assess  the  overall  effectiveness  of  C2. 

•  Established  objectives  for  the  assessment  must 
directly  link  to  the  MoMs. 

•  Selection  of  MoMs  should  consider  assumptions, 
constraints,  models,  tools,  scenarios,  other  elements 
of  the  analytic  plan,  and  assessment  processes. 

•  Identify  selected  MoMs,  their  thresholds  and 
standards,  their  means  of  collection,  their 
relationship  to  the  assumptions,  and  their 
imposed  constraints  in  the  assessment. 
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•  A  detailed  review  of  reliability  and  validity  for  the 
selected  measures  will  determine  a  level  of 
confidence  for  each  MoM. 

•  The  analyst  must  establish  and  measure  control 
variables  to  correlate  MoMs  in  a  spectrum  of 
multiple  scenarios. 

•  The  analyst  must  pay  particular  attention  to 
measurements  related  to  the  human  element, 
since  variations  in  measurements  may  well 
cause  unacceptable  levels  of  uncertainty. 

•  Analysis  of  uncertainties  and  measures  of 
central  tendency  and  dispersion  are  significant 
for  C2  issues. 

•  For  C2  acquisition  analyses,  measures  should  be 
generated  in  parallel  with  system  development, 
so  they  can  be  used  as  standards  for  system 
tests  and  operations. 


18 


NATO  Code  of  Best  Practice  for  C2  Assessment 


Environment 


Human  and  Organizational  Factors 

The  human  dimension  largely  distinguishes  C2 
analysis  from  other  military  operations  analysis.  C2 
analysis  must  deal  with  distributed  teams  including 
military,  interagency,  coalition  and  other  non-state 
actors  operating  under  stress  and  their  varying 
decisionmaking  behaviours.  In  OOTW,  particular 
attention  must  be  paid  to  the  behaviour  of  and 
interaction  with  non-military  organizations,  political 
groups,  and  amorphous  groups  such  as  crowds  and 
refugees.  Thus,  the  formulation  of  the  problem  and 
the  development  of  solution  strategies  cannot  be 
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completed  without  explicit  consideration  of  both 
human  and  organization  issues. 

•  Human  and  organizational  factors  must  be 
considered  as  part  of  structuring  the  problem, 
selecting  MoMs,  defining  scenarios,  developing 
solution  strategies,  and  selecting  methods  and 
tools,  and  they  should  be  reviewed  throughout 
the  entire  analytical  process. 

•  The  assessment  team  must  include,  or  have 
access  to,  experts  from  organizational  science  and 
the  various  human  science  disciplines  (such  as 
cultural  anthropology,  demography,  sociology, 
social  and  individual  psychology,  political  science). 

Human  Factors 

Human  factors  of  interest  fall  into  three  major  categories: 

•  Human  behaviour  related  to  performance  degradation 
and  as  a  consequence  of  social  interaction  among 
individuals  or  members  of  a  group; 

•  Decisionmaking  behaviour  (cognitive  questions) 
including  the  cognitive  complexity  of  issues  and 
the  capacities  of  the  commanders  and  other 
decisionmakers  of  interest; 

•  Command  style. 

Human  performance  depends  on  psycho-physiological 
variables  (e.g.,  stress,  sleep  deprivation,  hunger,  and 
alertness)  and  on  ergonomic  and  external  factors.  Individual 
and  group  behaviour  is  the  result  of  social  processes  and 
factors  (e.g.,  fear,  morale,  values),  and  the  cultural, 
educational,  and  religious  background  of  individuals. 
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•  Any  time  human  performance  and/or  behaviour 
are  at  issue,  parameters  and/or  models  are 
needed  to  reflect  those  issues.  Unless 
specialists  can  provide  valid  parameters  from 
work  in  other  contexts  or  from  field  experience  in 
OOTW,  some  experimentation  may  be 
necessary  and  appropriate  to  develop  them. 

•  Decisionmaking  that  is  automatable,  contingent, 
rule,  or  algorithmically  based  can  be  modelled 
directly,  but  time  requirements  and  error 
representation  must  be  incorporated  if  humans 
are  involved  in  the  actual  process. 

•  Complex  decisions  (e.g.,  courses  of  action  in 
response  to  events  in  the  operations  space)  are 
best  being  modelled  with  “human  in  the  loop” 
techniques.  Closed  form  techniques  for 
modelling  complex  decisions  are  still  in  the 
experimental  stage. 

•  Accounting  for  differences  in  command  styles 
and  how  they  affect  military  decisionmaking  is  of 
special  importance  in  OOTW  which  tend  to  be 
multinational  coalition  operations.  They  may  be 
reflected  by  attributes  such  as  the  background  of 
commanders,  their  field  experience,  risk 
attitudes,  and  organizational  and  orders  style. 

Organizational  Factors 

Organizational  design  reflects  the  interaction  among 
the  tasks  to  be  done,  the  people  to  do  them,  and 
the  systems  and  tools  that  support  those  people. 
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The  principal  differences  between  organizations  are 

related  to  structure,  function  and  capacity. 

Structural  differences  include: 

•  Number  of  echelons  in  the  command  structure; 

•  The  span  of  control  for  nodes  in  the  command 
structure;  and 

•  The  pattern  of  (formal  and  informal,  permanent 
and  transitory)  linkages  between  nodes 
(hierarchical  tree,  spokes  of  a  wheel,  multi- 
connected,  networked). 

Functional  differences  include: 

•  The  distribution  of  responsibility  (location  of 
functional  activities  such  as,  e.g.,  intelligence, 
logistics,  civil  military  cooperation  (CIMIC); 

•  The  distribution  of  authority  (ideally  co-located 
with  responsibility); 

•  Functional  specificity  (of  combat,  support,  or 
service  capabilities)  versus  integrated 
capabilities  (mission  tailored  taskforces);  and 

•  Degree  of  ambiguity  in  command  relationships. 

Capacity  differences  are  related  to: 

•  Personnel  (quality,  training,  experience); 

•  Information  and  Communications  systems  and 
architectures;  and 

•  C2  field  training  and  operational  experience. 
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A  systematic  approach  using  a  hypothesis-testing 
logic  and  aided  by  organisation  theory  expertise 
should  be  used  for  addressing  organizational  issues 
in  C2  assessment  because  of  the  large  number  and 
many  indirect  effects  of  organizational  variables. 

Scenarios 

Scenarios  provide  the  context  for  the  conduct  of  the 
operational  analysis  and  bound  the  arena  of  the 
analysis.  Scenarios  consist  of  several  static 
dimensions  shown  in  Table  1  and  include  the  dynamic 
evolution  of  events  in  time.  Operational  scenarios  detail 
threats,  orders  of  battle,  tactics,  rules  of  engagement 
and  courses  of  action,  deployments,  reserves, 
adversary  forces,  and  non-combatants. 

Attributes 

Several  prerequisites  are  essential  before  using 
scenarios  for  C2  analysis: 

•  Approval:  the  analyst  should  strive  for  the 
creation  of  a  family  of  scenarios  consistent  with 
high-level  guidance  and  policy; 

•  Breadth:  scenarios  should  reflect  those  factors 
that  are  hypothesised  to  have  a  significant 
impact  on  C2  issues; 

•  Capability:  scenarios  should  stress  C2 
capabilities,  including  human  and  organisational 
factors  (military  and/or  civilian)  where  appropriate; 

•  Credibility:  scenarios  should  include  logical 
assumptions  about  the  problem  under  analysis;  and 
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Table  1 .  The  Static  Dimensions  of  the  Scenario  Framework 
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•  Plausibility:  scenarios  should  represent 
plausible  situations. 

Principles 

•  Analysts  need  to  use  multiple  scenarios  and 
vignettes  to  cover  or  sample  the  interesting  problem 
space  since  no  single  scenario  is  sufficient. 

•  Analysts  should  explicitly  identify  and  describe 
the  scenarios  prior  to  the  execution  of  a  study.  It 
is  best  practice  to  revisit  the  scenario  definition 
periodically  during  the  conduct  of  the  study. 

•  For  coalition  C2  assessments,  scenarios  should 
be  developed  or  adapted  by  teams  with 
representatives  from  all  participating  nations. 

•  Scenarios  should  reflect  C2  organizations  and 
infrastructure  (including  human  issues), 
processes,  and  systems  relevant  to  the  analysis. 

•  Scenarios  must  consider  mission  scope, 
levels  of  hierarchy,  and  data  flow 
aggreg  ration/disaggregation. 

•  Key  scenario  assumptions  and  constraints 
should  be  identified  and  documented. 

•  Boundaries  of  the  scenario  spaces  should  be 
defined  by  the  problem  being  analyzed. 

•  Scenarios  should  reflect  the  factors  that  have 
significant  impact  on  C2,  stress  C2  issues,  are 
credible  to  the  military,  and  are  credible  in  terms 
of  civil-military  objectives. 
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Methods  and  Tools 

This  section  considers  the  best  methods  and  tools 
(quantitative  or  qualitative)  for  assessing  C2 
processes,  performance  and  effectiveness.  The 
section  covers  methods  and  tools  used  for  analysis, 
training  or  operations,  each  of  which  has  different 
requirements.  Available  methods  can  be  categorised 
into  several  classes,  namely:  data  collection/ 
generation,  data  organisation/relationship,  solving  and 
support.  Table  2  illustrates  this  categorisation  for  a 
sample  of  methods. 

A  key  challenge  for  C2  assessment  methods  is  to 
make  properly  quantified  linkages  between  MoP, 
MoCE,  MoFE,  and  MoPE.  All  C2  assessments 
require  a  high  level  of  creative  problem  structuring 
and  solving  in  making  these  linkages. 

Method  and  Tool  Selection 

The  following  issues  must  be  considered  when 
selecting  methods  and  tools  for  C2  assessments. 

•  The  inherent  complexity  of  C2  assessment 
problems,  which  calls  for  an  orchestrated  set  of 
complementary  tools  to  cover  the  wide  range  of 
variables  involved  (including  hierarchies  or 
federations  of  models  rather  than  dealing  with  all 
issues  in  a  single  model). 

•  The  requirement  to  explore  a  wide  range  of 
scenarios  and  still  represent  C2  processes  and 
performance  effectively. 
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Data  Data  Data 

_  ..  „  „  “  Organization/ 
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Army  C2  Evaluation 
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Miib-Criteria  Decision 
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Regression  Analysis 

Bayesian  Networks 

♦ 

• 
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Programming 
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Heuristic  Search 
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Genetic  Algorithms 

• 

• 

Project  Management 
Tools 

■ 

Data  Analysis 

• 

Geographical 
Information  Systems 

• 

Visualization 

• 

• 

Databases 

• 

• 

Checklists 

• 

Table  2.  Examples  of  Methods  and  Tools  Categorized  by  Use 


•  The  appropriate  treatment  of  human  issues  in 
models  and  the  balance  of  realism  against 
possible  sources  of  uncertainty  in  the  use  of 
“human-in-the-loop”  techniques. 

•  The  relative  advantages  of  stochastic  and 
deterministic  models  (depending  on  the  nature  of 
the  assessment  problem  and  other  elements  of 
the  solution  approach). 

•  The  appropriate  balance  for  the  representation  of 
friendly,  adversary  and  other  forces. 
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•  The  challenge  of  VV&A  and  the  development 
of  trust  in  methods  and  tools,  especially  in 
novel  applications  areas  such  as  unfamiliar 
OOTW  situations. 

Best  practice  for  the  application  of  analysis  tools 
for  C2  assessment  is  still  emerging,  but  the  following 
criteria  should  be  used  when  selecting  models. 

•  Functionality-based  criteria  (including  resolution, 
completeness/scope,  functionality,  explicitness, 
the  ability  to  generate  required  MoM,  validity, 
and  accreditation). 

•  Performance-related  criteria  (including 
responsiveness,  simplicity,  preparation/use  time, 
data  availability,  interoperability  with  other  tools, 
resource  requirements  and  credibility). 

Utility  of  Different  Types  of  Methods 
and  Tools 

•  The  use  of  model  federations,  particularly  with  an 
object-oriented  approach,  encourages 
development  using  holistic  and  evolutionary 
principles,  under  which  modellers  should  capture 
a  complete  model  of  the  process,  including  parts 
whose  representation  is  still  unclear  (ready  for 
improvement  as  understanding  develops). 

•  Agent-oriented  modelling  favours  the  capture  of 
the  cognitive  nature  of  command  tasks.  This  is 
important  in  establishing  the  linkage  between 
MoP  and  MoFE,  but  is  constrained  by  the  difficulty 
in  representing  human  performance,  command 
styles  and  organisational  relationships. 
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•  Very  fast-running  models  can  be  used  as 
screening  tools  during  problem  formulation  to 
scan  the  problem  space,  allowing  identification  of 
areas  of  concern  for  deeper  analysis. 

•  The  model-test-model  or  model-exercise-model 
processes  should  be  considered  where  the 
assessment  problem  requires  generation  of  new 
data  to  validate  existing  models  of  C2  factors. 
Assessment  teams  should  be  aware  of  the  latest 
advances  in  fast,  agile  modelling  and  exploit  them. 

Directions  for  Future  Development 

The  following  ideas  should  be  considered  in  new 
model  developments  (taking  account  of  the 
appropriate  level  of  model  aggregation). 

•  Understanding  of  adversary  intent  can  be 
represented  by  having  a  set  of  prescribed  options; 

•  HQs  with  a  local  “picture”  in  each  should  be 
explicitly  represented  to  allow  different 
perception-based  behaviours  to  be  represented, 
facilitating  the  study  of  factors  such  as 
deception,  shock,  and  surprise; 

•  Information  should  be  represented  as  a 
commodity  as  follows: 

•  a  realistic  flow  of  information  around  the 
operational  environment; 

•  the  collection  of  information  from  multiple 
sources  and  the  tasking  of  information 
collection  assets; 
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•  the  processing  of  information; 

•  C2  systems  as  entities  on  the  battlespace/ 
operational  space; 

•  unit  perceptions  built,  updated,  and  validated 
from  the  information  available  to  the  unit 
through  its  information  systems; 

•  the  commander’s  decision  based  on  the 
unit’s  perception  of  the  operational  space; 

•  information  operations  actions  and  impacts 
across  all  sides  represented  in  the  model. 

Data,  Meta-data,  and  the 
Common  Data  Infrastructure 

While  the  value  of  data  for  an  individual  study  effort  is 
well  understood  by  the  analytic  community  at  large, 
the  aggregated  worth  of  data  is  still  undervalued  by 
many.  Data  can  be  described  as  the  fundamental 
elements  of  information  and  knowledge  that  comprises 
the  corporate  whole  -  consequently  its  aggregated 
value  particularly  when  addressed  in  a  context  larger 
than  an  individual  study  is  significantly  greater  than 
the  sum  of  the  parts.  Explicitly  dealing  with  this  issue 
resulted  in  data  receiving  its  own  chapter  in  the  code. 

Data  are  factual  information  organised  for  analysis. 

•  The  Data  Taxonomy  in  the  COBP  lays  out  a 
number  of  types  of  data  including  broad 
categories  of  sources  that  will  be  of  interest  to 
the  analyst.  The  ability  to  determine  the  needed 
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data  and  the  ability  to  assemble  or  collect  these 
data  will  constrain  the  solution  strategy. 

•  A  major  challenge  is  to  find,  organise,  verify, 
process,  and  convert  source  data  into  the  data 
needed  for  the  study.  The  team  needs  to  know 
(a)  what  data  are  needed  in  which  structure;  (b) 
who  owns  these  data;  (c)  security  issues;  and  (d) 
costs  to  buy,  collect,  or  generate  data. 

•  If  the  data  are  not  available  and  can  neither  be 
aggregated  nor  derived  from  the  available 
sources,  it  is  good  practice  to  use  the  knowledge 
of  subject  matter  experts  to  generate  the 
necessary  data. 

Meta-data  are  “information  about  information,”  which 
are  used  to  document  the  data  and  related  issues. 

•  The  source  of  the  data,  the  reliability,  and  assorted 
assumptions  and  constraints  must  be  captured  in 
standardized  meta-data  sets  assigned  to  the  data 
sets  comprising  the  needed  information. 

•  Data  that  are  collected  without  adequate 
documentation  are  frequently  viewed  as  suspect 
or  unusable.  To  avoid  having  good  data  thrown 
away  due  to  the  lack  of  documentation, 
acceptable  community  standards  for 
documentation  must  be  employed.  This 
information  has  to  be  captured  in  standardized 
meta-data  sets. 

•  The  initial  data  available  will  often  be  vague, 
uncertain,  incomplete,  and  contradictory.  On  the 
other  hand  analysts  prefer  data  to  be  sharp, 
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certain,  complete,  and  consistent.  The 
assumptions  inherent  in  accomplishing  this 
transformation  must  be  captured  in  standardized 
meta-data  sets. 

A  Common  Data  Infrastructure  facilitates  reuse 
of  data.  It  is  based  on  agreed  standards  for  data 
formats,  meta-data,  and  documentation. 

•  Data  are  central  to  the  conduct  of  good 
assessments.  Adhering  to  data  engineering 
principles  contributes  to  data  reuse.  Archiving  of 
data  in  retrievable  form  using  standardized  meta¬ 
data  sets  is  one  essential  component  to  facilitate 
data  reuse. 

•  As  the  data  being  used  today  by  the  analysts  will 
be  the  data  needed  tomorrow  by  systems 
engineers,  decisionmakers,  and  commanders  for 
their  operations,  it  is  good  practice  to  align  the 
standardization  processes  and  respective  tool 
sets  between  the  analytical  and  operational 
communities  in  order  to  facilitate  the  transfer  of 
data  between  those  communities. 

Risk  and  Uncertainty 

Definition 

Risk  is  commonly  defined  as  the  possibility  of 
suffering  harm  or  loss.  Uncertainty  can  be  defined 
as  an  inability  to  determine  a  variable  or  system  state 
or  predict  its  future  evolution. 
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There  are  risks  associated  with  the  uncertainties  in 
decisionmaker’s  situation  that  are  the  subject  of  the 
assessment  and  there  are  risks  related  to  the  conduct 
of  the  assessment  itself.  Using  the  COBP  should  help 
minimise  the  risks  involved  in  C2  studies,  but  the 
following  risk  and  uncertainty  issues  remain  and  are 
intrinsic  to  any  C2  study. 

Scope  of  Risk  and  Uncertainty 

•  The  explicit  treatment  of  risk  and  uncertainty  is 
best  practice  in  all  studies  and  of  particular 
importance  in  C2  assessment.  It  is  advisable  not 
to  skip  risk  analysis,  even  when  time  and 
resources  are  limited,  and  this  should  be 
stressed  to  decisionmakers  and  study  sponsors. 

•  Perceptions  of  risk  and  uncertainty  can 
substantially  differ  from  objective  assessments  and 
this  should  be  explicitly  considered.  Analysts 
should  find  out  how  study  sponsors  perceive  risks. 

•  A  thorough  understanding  of  study  variables  is 
essential  for  effective  treatment  of  uncertainty. 
OOTW  studies  typically  have  less  well-formed 
quantitative  factors  and  more  qualitative  factors. 

•  The  assessment  team  must  be  aware  of  sources 
of  uncertainty  in  all  aspects  of  a  study,  including 
those  related  to  the  assumptions  and  limitations 
inherent  in  parameter  values,  models,  scenarios, 
data  structures,  and  the  boundaries  used  to 
scope  and  focus  the  study.  Humans  involved  in 
the  assessment  process  make  assumptions  that 
should  be  identified,  documented  and  analysed. 
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•  In  C2  assessments,  analysts  need  to  be 
particularly  alert  to  the  possibility  of  chaotic 
behaviours  arising  from  dynamic  interactions  of 
human  and  organisational  factors,  and 
discontinuous,  non-linear  divergence  arising  from 
the  multiple  options  for  choice  of  course  of  action. 

Treatment  of  Risk  and  Uncertainty 

•  Where  uncertainties  cannot  be  reduced  by 
acquiring  more  information,  multi-factorial 
sensitivity  analysis  should  be  used  to  establish  the 
regions  for  which  the  results  are  valid  and  to  isolate 
those  factors  that  may  introduce  uncertainty. 

•  Sensitivity  analysis  should  not  only  deal  with 
statistical  variance  but  should  also  consider 
qualitative  variations  in  models,  perspectives  and 
assumptions.  A  range  of  analytic  tools  relevant 
to  sensitivity  analysis  is  identified  in  the  full  text 
of  the  COBP. 

•  Checklists  and  structured  appraisal  help  to 
maintain  study  rigour,  but  neither  is  a  substitute 
for  critical  thinking  in  the  specific  study  context. 

•  The  team  should  expect  a  complex  and  partly 
hidden  set  of  risks  to  C2  studies  and  serious 
efforts  should  be  made  to  illuminate  them  in  an 
explicit  risk-based  analysis  with  portfolio-based 
solutions.  Such  analysis  needs  metrics  for  risk 
and  failure  as  well  as  MoMs. 

•  The  assessment  team  should  make  a  complete 
list  of  risks  and  treat  them  in  appropriate  detail  in 
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a  risk  register  for  the  study.  A  generic  risk  register 
has  been  produced  to  support  the  COBP. 

Communication  of  Risk  and  Uncertainty 

Communication  of  uncertainty  and  risk  is  particularly 
important  for  C2  assessment  products.  Such 
communication  must  take  account  of  the  complexities 
of  the  subject  and  human  limitations  in  understanding 
risk  and  uncertainty. 

A  continuing  dialogue  with  stakeholders  about 
uncertainty  will  facilitate  common  understanding, 
including  the  impact  of  uncertainty  on  the  robustness 
of  conclusions  and  methods  for  mitigating  them. 

Products 

Assessment  products  communicate  results  to 
decisionmakers  and  stakeholders,  establish  the 
credibility  of  the  effort,  and  provide  a  lasting  record  of 
the  project.  Typical  C2  assessment  products  include 
the  Study  Plan,  Periodic  Status  Reports,  the  Project 
Journal  and  a  Final  Report  and  Briefing,  which  include 
a  variety  of  supporting  data  and  documents. 

•  The  Study  Plan,  a  living  document  kept  current 
throughout  the  assessment,  explains  the  problem 
under  analysis,  the  solution  strategy,  the  tasks 
involved,  and  how  they  fit  together.  This  will 
include,  as  appropriate,  plans  necessary  for  the 
assessment,  such  as  data  collection  and  data 
analysis  plans.  The  elements  of  a  Study  Plan  are 
enumerated  in  the  Solution  Strategy  section. 
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•  Periodic  Status  Reports  describe  the  state  of  the 
effort  over  time  and  can  be  assembled  to  create 
a  record  of  the  work  accomplished,  problems 
encountered,  results  from  peer  reviews,  and  the 
adjustments  made  by  the  team. 

•  The  Project  Journal  records  interactions  with 
professionals  outside  the  core  team  and 
captures  the  analytic  assumptions  and  decisions 
made  by  the  assessment  team  (such  as  choice 
of  tools  and  selection  of  data  or  methods  for 
generating  data)  over  time. 

•  The  Final  Report  presents  the  results  of  the  effort 
(findings,  recommendations,  and  lessons 
learned)  and  incorporates,  as  necessary,  items 
from  the  other  key  documents.  It  will  also 
typically  include  appendices  that  provide 
supporting  detail  such  as  participants  in  the 
effort,  references,  glossary,  list  of  acronyms  and 
abbreviations,  as  well  as  technical  material  such 
as  data  collection  instruments. 

•  The  Final  Briefing,  which  typically  is  circulated 
more  broadly  than  the  Final  Report,  will  provide 
a  summary  of  the  most  important  aspects  of  the 
project,  including  the  problem  statement, 
solution  strategy,  research  accomplished, 
findings,  recommendations,  and  lessons 
learned.  Best  practice  requires  that  this 
document  be  richly  annotated. 

•  Data,  models,  supporting  scenario  materials,  and 
other  products  necessary  to  make  the  results 
credible  and  authoritative  or  that  may  be  of  value 
to  other  researchers  or  assessment  teams 
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should  be  archived  and  made  available  as 
broadly  as  possible. 

In  general,  all  the  products  should  be  circulated  as 
widely  as  possible  so  that  others  can  benefit  from  the 
work  accomplished  and  redundant  projects  avoided. 
This  requires  that  specific  effort  be  made  to  downgrade 
or  sanitize  products  and  to  identify  those  who  would 
benefit  from  reviewing  them. 


'North  Atlantic  Treaty  Organisation  (NATO)  NATO  Code  of  Best 
Practice  for  C2  Assessment,  2002. 

2North  Atlantic  Treaty  Organisation  (NATO)  Annex  B  to  MC 
Guidance  for  Defence  Planning.  MC-299/5.  (1996) 

3North  Atlantic  Treaty  Organisation  (NATO)  Bi-MNC  C2  plan 
part  2 — Command  and  control  requirements.  (1 998). 


ANNEX  A 


Decisionmaker’s 
Temporal 
Question  List 


Attached  is  a  list  of  questions  that  have  been 
included  in  the  decisionmakers’  guide.  It  would 
be  prudent  if  the  analyst  is  prepared  to  respond  to 
these  questions. 

This  annex  contains  key  questions  that  a 
decisionmaker  should  ask  the  C2  assessment  team. 
These  questions  are  organized  temporally  according 
to  the  following  phases  of  a  study  (prior  to  the  study, 
at  initial  review,  after  first  iteration,  and  at  final  report). 

Prior  to  the  Study: 

•  Do  you  understand  what  decisions(s)  I  have  to 
make,  when  I  have  to  make  them,  and  the  context 
within  which  the  decision(s)  will  be  made? 

•  Do  you  need  any  information  or  authorization 
from  me? 

•  Who  will  be  on  the  study  team? 

•  Are  there  adequate  skills,  experience  present 
in  the  team? 
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•  For  OOTW  studies  in  particular,  are  there 
adequate  social  scientific  skills  in  the  team? 

•  Who  are  the  key  organizations/individuals  with 
whom  you  plan  to  interact  (e.g.,  stakeholders, 
data  providers,  review  team)? 

•  Have  you  coordinated  the  Terms  of 
Reference  with  them? 

•  How  do  you  plan  to  interact  with  them? 

•  Particularly  for  OOTW  studies,  how  will  you 
acquire  the  requisite  knowledge  of  the  culture/ 
historical  context? 

•  How  will  you  undertake  problem  formulation?  e.g., 

•  What  products  will  you  review/mine? 

•  What  methods  and  tools  are  applicable 

•  When  will  key  events  occur  (e.g.,  reviews, 
production  of  interim  products)? 

At  the  Initial  Review: 

•  What  do  you  perceive  the  “real”  issues  to  be? 

•  What  assumptions  do  you  plan  to  make  to 
scope  the  effort? 

•  What  do  you  plan  to  use  for 

•  High  level  MoMs? 

•  Scenarios  of  interest? 
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•  Have  you  identified  any  additional  organizations/ 
individuals  with  whom  you  plan  to  coordinate? 

•  How  do  you  plan  to  attack  the  problem? 

•  What  methodology  will  you  employ? 

•  How  will  you  treat  the  diverse  aspects  of  a 
mission  capability  package  (including  concept 
of  operations,  organization,  doctrine,  C2 
approach,  systems,  personnel,  facilities,  in 
other  words  everything  needed  to  field  a  real 
capability.) 

•  How  do  you  plan  to  address  organization/ 
human  issues? 

•  What  specific  methods  and  tools  will  you 
employ?  Why  do  you  think  they  are 
appropriate? 

•  What  data  will  you  employ?  Where  will  you  get 
them?  Why  do  you  think  they  are  appropriate? 
How  do  you  plan  to  make  the  data  accessible  to 
others?  How  do  you  plan  to  depict  the  results  of 
the  study? 

At  First  Iteration: 

•  What  specific  MoMs  were  selected?  What 
relationships  were  established  among  the 
MoMs? 

•  What  range  of  scenarios  were  selected?  Why? 

•  What  plans  do  you  have  to  illuminate  uncertainty/ 
sensitivity? 
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•  What  feedback  did  you  receive  from  the 
independent  review  team?  What  steps  did  you 
take  to  respond  to  it? 

•  What  do  you  plan  to  do  on  subsequent  iterations? 

•  Use  additional  tools? 

•  Consider  additional  scenarios,  assumptions? 

•  Modify  assessment  boundaries? 

At  Final  Report: 

•  What  are  the  major  findings,  recommendations? 

•  What  are  the  key  points  of  uncertainty/sensitivity? 

•  What  issues  were  not  addressed  that  should  be 
treated  in  subsequent  assessments? 

•  What  key  lessons  did  you  learn  with  respect  to 
methods,  tools,  and  data? 

•  What  steps  are  you  going  to  take  to  disseminate 
key  insights,  products  to  the  broader  community? 


Are  there  any  voids/issues  that  warrant 
further  research? 


